Dynamic load balancing using quality/loading bands

ABSTRACT

Methods and apparatus for. An example method includes determining, by a network device, respective quality metrics for each of a plurality of members of an aggregation group of the network device, the respective quality metrics representing respective data traffic loading for each member of the aggregation group. The example method further includes grouping the plurality of aggregation members into a plurality of loading/quality bands based on their respective quality metrics. The example method also includes selecting members of the aggregation group for transmitting packets from a loading/quality band corresponding with members of the aggregation group having lower data traffic loading relative to the other members of the aggregation group.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. §119 to U.S.Provisional Patent Application Ser. No. 61/237,661, filed Aug. 27, 2009.This application is also related to co-pending U.S. patent applicationSer. No. 12/780,493, entitled “DYNAMIC LOAD BALANCING” and co-pendingU.S. patent application Ser. No. 12/780,524, entitled “DYNAMIC LOADBALANCING USING VIRTUAL LINK CREDIT ACCOUNTING”, both to the sameinventors and filed on a date even with the present application. Thedisclosures of U.S. Provisional Patent Application Ser. No. 61/237,661,U.S. patent application Ser. No. 12/780,493 and U.S. patent applicationSer. No. 12/780,524 are incorporated by reference herein in theirentirety.

TECHNICAL FIELD

This description relates to processing of packets in data networks.

BACKGROUND

Use of network data communications for commercial and non-commercialapplications continues to grow at an extremely rapid pace. This growthin the use of data communications is a driving factor for thedevelopment and implementation of data networks that provide increasedbandwidth and reliability in order to meet the ever growing demand ofconsumers (commercial and non-commercial) for such data communicationservices.

One approach that is used to provide such high-bandwidth datacommunication and reliable data communication services is the use ofaggregation groups for communicating data between nodes (networkdevices) in data networks. For instance, an aggregation group may beimplemented using multiple data communication links between two networkdevices on a data network where the members of the aggregation groupfunction together to communicate data between the two network devices.Such network devices may be directly connected, or may be connectedthrough additional network devices. Depending on the direction of databeing communicated between such devices, one device may operate as adata source and one device may operate as a data destination. The datacommunication links of an aggregation group (the members) between thesource and destination devices may take the form of physical links or,alternatively, may take the form of virtual links. Virtual links, as theterm indicates, may be implemented by defining a plurality of logicallinks. The plurality of logical links may then be implemented using oneor more physical links, depending on the particular embodiment. Themultiple data communication links/paths (whether physical or virtuallinks) may provide for increased aggregate bandwidth (e.g., for anaggregation group as compared to a single member of the aggregationgroup) and may also provide for improved reliability as such aggregationgroups provide multiple communication links/paths between a source anddestination.

Aggregation groups may be implemented in a number of fashions. Forinstance, an aggregation group may be implemented using Layer-3 (L3)Equal Cost Multi-Path (ECMP) techniques. Alternatively, an aggregationgroup may be implemented as a link aggregation group (LAG) in accordancewith the Institute of Electrical and Electronics Engineers (IEEE)802.3ad standard. As yet another alternative, an aggregation group maybe implemented as a Hi-Gig trunk (e.g., using four 2.5gigabit-per-second (Gbps) data traffic links to form a 10 Gbps trunk).Of course, other approaches for implementing an aggregation group may beused.

In order to assign data flows to the members of an aggregation group,hash-based load balancing techniques are used to assign each data flowthat is processed by a network device to a respective member of theaggregation group (e.g., one of the individual physical or virtuallinks). A data flow may be defined in a number of ways. For instance, adata flow may be defined as a flow that includes data packets that areassociated with an individual application and/or user, which may bereferred to as a micro-flow. Alternatively, a data flow may be definedas a flow that includes packets from different micro-flows that hash tothe same value. Such a data flow may be termed a macro-flow.Accordingly, macro-flows may include multiple micro-flows. For purposesof clarity, the term “flow” or “data flow,” as used herein, may refer toany data flow (e.g., micro-flows or macro-flows) where packets includedin the flow have a common set of packet fields and, therefore, hash tothe same value using those common values to generate hash values, evenif the packets in a data flow are not associated with a single user orsingle application. Such packet fields may include, for example, asource address, a destination address, a packet priority (such asindicated in respective p-bits of the packets), a protocol identifier,among a number of other characteristics.

Using hash based load-balancing, a fixed set of packet fields may bedefined for use in member assignment. When a packet arrives, the packetmay indicate an aggregation group that is to be used to communicate thepacket. A hash value is then generated based on the values of the fixedset of fields in each of the packets (e.g., using an XOR hash or acyclic redundancy check (CRC) hash). A member of the aggregation groupindicated in the packet is then assigned based on the hash value using amathematical transformation of the hash value. For instance, a modulooperation may be performed on the hash value, such as, Hash Value mod k,where k is the number of members of the designated aggregation group.

While such hash based approaches may provide for random statisticaldistribution of flows across aggregate members, such approaches havecertain drawbacks. For instance, if there are large differences in theamount data being communicated in a flow or flows that are assigned toone aggregate member as compared to the amount of data beingcommunicated by a flow or flows that are assigned to another aggregatemember, this situation may result in a significant imbalance in theamount of data traffic being communicated by members of an aggregationgroup. Such imbalances may result in data congestion, packets beingdropped and, therefore, inefficient use of available bandwidth. Forinstance, one member of an aggregation group could be oversubscribed(receive more data than it can transmit in a give time) while othermembers of the aggregation group are idle or carry very little datatraffic.

SUMMARY

A method and/or apparatus for dynamic load balancing of data traffic ina data network, substantially as shown in and/or described in connectionwith at least one of the figures, as set forth more completely in theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a data network in accordance withan example embodiment.

FIG. 2 is a block diagram illustrating a network device in accordancewith an example embodiment.

FIG. 3 is a block diagram illustrating an approach for determining if adata flow is eligible for load balancing reassignment in accordance withan example embodiment.

FIGS. 4A and 4B are diagrams illustrating an example flow set table andan approach for indexing the flow set table in accordance with anexample embodiment.

FIGS. 5A and 5B are tables illustrating quality metric quantization inaccordance with an example embodiment.

FIG. 5C is a diagram illustrating a quality mapping table based on thequality metrics of FIGS. 5A and 5B in accordance with an exampleembodiment.

FIG. 6 is a graph illustrating quality/loading bands in accordance withan example embodiment.

FIG. 7 is a diagram illustrating an approach for random selection of anaggregate member in accordance with an example embodiment.

FIG. 8 is a flowchart illustrating a method for dynamic load balancingin accordance with an example embodiment.

FIG. 9 is a flowchart illustrating a method for indexing a flow settable in accordance with an example embodiment.

FIG. 10 is a flowchart illustrating a method for determining a qualitymeasure in accordance with an example embodiment.

FIG. 11 is a flowchart illustrating a method for generatingquality/loading bands for an aggregation group and selecting anaggregate member using the quality/loading bands in accordance with anexample embodiment.

FIG. 12 is a flowchart illustrating a method for dynamic load balancingfor virtual links of an aggregation group in accordance with an exampleembodiment.

FIG. 13 is a diagram illustrating an embodiment of quality/loading bandsin accordance with an example embodiment.

FIG. 14 is a flowchart illustrating a method for generatingquality/loading bands for an aggregation group and selecting anaggregate member using the quality/loading bands in accordance with anexample embodiment.

FIG. 15 is a flowchart illustrating a method for generatingquality/loading bands for an aggregation group and selecting anaggregate member using the quality/loading bands in accordance with anexample embodiment.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating a data network 100 in accordancewith an example embodiment, which may be used to implement thetechniques described herein. The network 100 may be part of a largernetwork, such as, for example, a local area network, a wireless network,a public network (such as the Internet), or a number of other datanetwork configurations. The network 100 includes a source network device(source) 102 and a destination network device (destination) 104. In thisexample, data packets included in one or more data flows may be sentfrom the source 102 to the destination 104. The source 102 and thedestination may take a number of forms and may be used to implementdifferent layers of a packet communication protocol. For example, thesource 102 and the destination 104 may take the form of networkswitches, routers, network interface cards, or other appropriate datacommunication device.

The designation of the source 102 and the destination 104 in thisexample is by way of illustration. In other embodiments, the source 102and the destination 104 may be reversed. In still other embodiments,network devices (such as the source 102 and the destination 104) may actas a source for outgoing data flows departing from the respective deviceand act as a destination for incoming data flow arriving into therespective device (e.g., data communication between network devices maybe bi-directional).

As shown in FIG. 1, the network 100 also includes an aggregation group106 that may be used to communicate data between the source 102 and thedestination 104. The aggregation group 106 may include a plurality ofphysical links between the source 102 and the destination 104.Alternatively, the aggregation group 106 may include a plurality ofvirtual links that are implemented using one or more physical linksbetween the source 102 and the destination 104. In the example shown inFIG. 1, the aggregation group 106 includes N aggregate members. As shownin FIG. 1, these aggregate members may take the form of Path_A 108,Path_B 110 through Path_N 112. Each of the paths 108-112 may be aphysical link or a virtual link, as appropriate for the particularembodiment. For instance, the paths 108-112 of the aggregation group 106may take the form of L3 ECMP next hop links, LAG communication links orHiGig communication links, as some examples. Of course, other types ofaggregation groups may be used.

Selection of members of the aggregation group 106 for data flows sentfrom the source 102 to the destination 104 may made using the techniquesdescribed herein. Briefly, selection of a member of the aggregationgroup 106 (Path_A 108, Path_B 110, etc.) for communicating a data flowmay be based, for example, on (i) the loading of each aggregate memberwhen the flow is being assigned, (ii) changes in loading conditionsamong the aggregate members and (iii) whether a given flow has been idlefor a long enough period of time that it may be considered as a “new”flow and assigned to a different aggregate member without risk ofpackets arriving out of order. Such re-assignment may be done by takingthe skew of packet transmission times for each of the paths intoaccount, such as is the manner described below with respect to FIG. 3.

FIG. 2 is a block diagram illustrating a network device 200 inaccordance with an example embodiment. The network device 200 may beused to implement the data traffic load balancing techniques describedherein. For instance, the network device 200 may be configured todynamically assign data flows to members of an aggregation groupassociated with it. Such member assignments may take into account therelative loading of the aggregation group's members (e.g., physicallinks or virtual links) and/or changes in the loading conditions of theaggregation group's members. Further, the network device 200 may beconfigured to re-assign data flows from one aggregate member to anotheraggregate member when those flows have been inactive for a sufficientperiod of time so as to prevent re-ordering of packets of the data flowsupon arrival at a next hop network device or a destination networkdevice.

As shown in FIG. 2, the network device 200 may receive a packet 202 at adata port 204. After the packet 202 is received at the data port 204, aflow identification (flow ID) module 206 may process the packet 202 todetermine a flow ID for a data flow to which the data packet 202belongs. An example approach for determining a flow ID is brieflydescribed here and is shown in further detail in FIG. 4. The flow ID maybe determined from the packet 202 by first generating a hash value fromthe packet 202 and using the hash value to determine a correspondingindex for the data flow, such as by using an appropriate mathematicaltransformation (e.g., a modulo operation). In an example embodiment, thehash value may be determined from a set of packet fields that remainfixed for a given data flow (micro-flow or macro-flow). Depending on theparticular situation, a flow ID may correspond with a single active dataflow (e.g., a micro-flow) or may correspond with multiple data flow thatare active concurrently (e.g., a macro-flow).

The determined index may be combined with one or more offsets todetermine the flow ID, where the flow ID corresponds with an entry in aflow set table, such as the example flow set table illustrated in FIG. 4and described further below. Such offsets may include, for example, afirst offset corresponding with an aggregation group identified in thepacket 202. Additionally, the offsets may include a second offset thatcorresponds with a priority of the packet (and the corresponding dataflow). The flow ID module may combine the determined index with the oneor more offsets to produce the flow ID for the packet 202.

In the network device 200, a flow state module 208 may be configured todetermine a state of the data flow with which the packet 202 isassociated. The flow state module 208 may make this determination, forexample, by examining an entry in the flow set table that correspondswith the flow ID of the packet 202 as determined by the flow ID module206. Based on the corresponding flow set table entry, the flow statemodule 208 may determine that the network device 200 has not previouslyprocessed any packets having the determined flow ID for the packet 202(new flow). In this situation, the flow set table entry may be invalidor contain initialization values (e.g., all zeros or ones).Alternatively, the flow state module 208 may determine, from the flowset table entry, that the packet 202 is associated with a data flow thathas been previously observed by the network device 200 and is activelytransmitting data (active flow). In this situation, the network device200 should continue using the most recent aggregate member assignment(as indicated by the flow set table entry) to prevent re-ordering ofpackets. As yet another alternative, the flow state module 208 maydetermine that the packet 202 is associated with a data flow for whichthe network device 200 has previously processed packets, but that thedata flow has been inactive for a sufficient period of time (such asdiscussed with respect to FIG. 3 below) that assigning the data flow toa different aggregate member would be unlikely to result in packetreordering once the packet 202 reaches a destination network device(next hop device) for the packet 202 (inactive flow).

In the network device 200, once the flow state module 208 determines thestate of the data flow associated with the packet 202, such as in thefashion described above, a member assignment module 210 may determinewhich member of an aggregation group of the network device 200 should beused to communicate the packet 202 to its next destination. Aspreviously discussed, the aggregation group members may comprisephysical links or virtual links, depending on the particular embodiment.

If the data flow associated with the data packet 202 is determined to beeligible for member assignment (e.g., has not been previously seen) orreassignment (e.g., has been inactive sufficiently long enough toprevent re-ordering), the network device 200 may assign the data flowassociated with the packet 202 to an aggregation group member using oneor more respective quality metrics associated with each of theaggregation group members, such as using the techniques describedherein. Once a member assignment has been made, the network device 200(e.g., the member assignment module 210) may be configured to makecorresponding entries in a flow set table, where the entries indicatethe assigned member as well as a timestamp indicating a time at whichthe packet 202 is processed. Such timestamps may be used to determinewhether data flows processed by the network device 200 have beeninactive long enough to allow for reassignment without significant riskof causing packet reordering.

If a data flow associated with the data packet 202 is not eligible forassignment or reassignment (e.g., has been previously seen and haspackets being actively transmitted), the network device 200 (e.g., themember assignment module 210) may be configured to use a previouslyassigned aggregation group member, such as indicated in a flow set tableentry associated with the data flow. Each time a packet associated withan active data flow is processed, the flow set table entry for that flowmay be updated with a new timestamp. Using such an approach, the delaybetween packets of a given data flow can be readily determined, wherethe determined delay between packets of the given data flow may be usedto determine whether the given data flow may receive a new assignmentwithout significant risk of packet reordering resulting.

After the assigned aggregation group member is determined (byassignment, reassignment or using an existing assignment), the networkdevice 200 may queue the data packet 202 in a respective data queue forthe assigned aggregation group member in a set of aggregate member dataqueues 212 for the aggregation group. The network device 200 may thencommunicate the data packet 202, in turn with other packets queued inthe aggregate member queues 212, to another network device, such as anext hop network device or a destination network device, using theassigned aggregation group member.

As shown in FIG. 2, the network device 200 may also include an imbalancedetection module 214. The imbalance detection module 214 may receiveinformation regarding member assignments for data packets from memberassignment module 210 and may also receive information regarding theamount of data that is queued for each member of an aggregation groupfrom the aggregate member queues 212. Based on the information receivedfrom the member assignment module 210 and the aggregate member queues212, the imbalance detection module 214 may determine one or morequality metrics for the aggregation group members, where the qualitymetrics are indicative of data traffic loading for each aggregatemember. These quality metrics may be communicated to the memberassignment module 210 and used by the member assignment module 210 whenmaking aggregation group member assignments and reassignments for dataflows being processed by the network device 200.

FIG. 3 is a block diagram illustrating a network 300 that may be used toimplement an approach for determining if a data flow is eligible forload balancing reassignment in accordance with an example embodiment. Inthe network 300, a data flow 302 arrives at a source network device 304.The source 304, in this example, has two aggregate members to which itcan assign the data flow 302, Path_A 308 and Path_B 310. For thisexample, it is presumed that Path_A 308 is initially selected as theaggregate member assignment for transmitting packets from the source 304to a destination 306. As shown in FIG. 3, an end-to-end delay for thePath_A 308 is 90 μs, while the end-to-end delay for the Path_B 310 is 60μs. In this example, the skew between the Path_A 308 and the Path_B 310is therefore 30 μs.

As indicated in FIG. 3, the data flow 304 has a packet arrival delay of50 μs between two sequential packets of the data flow 304. In thisinstance, the packet arrival delay of 50 μs is greater than the pathskew of 30 μs between the aggregate members of the network 300.Accordingly, in this situation, the flow 304 could be reassigned fromthe Path_A 308 to the Path_B 310 without significant risk thatreordering would occur at the destination 306.

In such approaches, the path skew can be estimated or measured (e.g.,using software). Such path skew measurements may be used to determine aninactivity threshold for use in determining whether data flows areeligible for aggregation group member reassignment. For instance, pathskew between the various paths of an aggregation group could be measuredor estimated. The largest path skew value (or the largest path skewvalue plus some margin) could then be used as an inactivity threshold.Using such an approach, if the arrival delay of packets of a given dataflow is greater than the inactivity threshold, the corresponding dataflow may then be eligible to receive a new aggregation group memberassignment. In yet another approach, packets of a given data flow may beconsidered eligible to receive a new aggregation group member assignmentunder all conditions.

In other approaches, path skew values between individual paths may beused for determining inactivity durations to determine whether or notdata flows are eligible for aggregation group member reassignment.Measurements or estimates of path skews (and the determination ofcorresponding inactivity thresholds) could be made dynamically (e.g.,during processing of data flows). Alternatively, measurements orestimates of path skews (and the determination of correspondinginactivity thresholds) could be made as part of initialization of acorresponding network device and statically set.

In an example embodiment, such inactivity thresholds may be used todetermine whether a data flow is active or inactive (as discussed above)for purposes of determining whether data flows are eligible foraggregation group member reassignment. For instance, determining that adata flow is inactive may include determining a difference between aprevious timestamp associated with the flow in a flow set table and atimestamp associated with the current packet. If the difference isgreater than an inactivity threshold, the flow would be considered to beinactive and eligible for reassignment to a new aggregate member. If,however, the difference is less than or equal to the inactivitythreshold, the flow would be considered to be active and, therefore, noteligible for reassignment.

FIGS. 4A and 4B are diagrams illustrating an example flow set table 410and an approach for indexing the flow set table 410 in accordance withan example embodiment. FIG. 4B, specifically, is a diagram of an exampleflow set table entry 412, such as may be included in the flow set table410 shown in FIG. 4A. The approaches illustrated in FIGS. 4A and 4B maybe implemented in, for example, the network device 200 illustrated inFIG. 2. Of course, the approaches illustrated in FIGS. 4A and 4B may beimplemented in conjunction with other network device configurations aswell.

In FIG. 4A, a packet 402 may be provided to an index and offsetdetermination module 408. In the approach illustrated in FIG. 4A, theindex and offset determination module 408 may be used to determine aflow ID, such as was previously described. As shown in FIG. 4A, thepacket 402 may include an aggregation group identifier (ID) 404 and apriority 406. The priority 406 may take the form of p-bits of the packet402, or may be included in the packet in another appropriate fashion. Asis known, the packet 402 may include a number of other elements, such asvarious header fields and a data payload, for example.

The index and offset determination module 408 may determine an indexvalue for the packet 402 based on one or more fields of the packet. Aspreviously discussed, a fixed set of packet fields that remain constant(produce the same index value) for a given data flow may be used todetermine the index for the packet 402. For instance, a hash value maybe calculated and/or one or mathematical operations may be performed onthe fixed set of packet fields to determine the index. In certainembodiments, the aggregation group ID 404 and/or the priority 406 of thepacket 402 may be used to determine the number of aggregation groupmembers in the corresponding aggregation group and/or priority set whendetermining the index. For instance, the number of members in acorresponding aggregation group and priority set of that aggregationgroup may be used in a modulo function to determine the index value forthe packet 402.

In the approach shown in FIG. 4A, a determined index value may becombined with two offset values to determine a flow ID that is used toindex the flow set table 410. For instance, in FIG. 4A, the index valuefor the packet 402 may be combined with an aggregation group offset anda priority set offset to determine the flow ID for the packet 402. Asshown in FIG. 4A, the flow set table 410 includes flow set entries foreach of three aggregation groups 1-3 and two priority sets 0 and 1within each aggregation group. In this example, the aggregation group 1may have an offset of 0, since the aggregation group 1 flow set entriesare at the top of the flow set table 410. The aggregation group 2 mayhave an offset of 5 as the first entry for the aggregation group 2 is atthe fifth location in the flow set table 410. Similarly, in the exampleshown in FIG. 4A, the aggregation group 3 may have an offset of 11 asthe first entry for the aggregation group 3 is at the eleventh locationin the flow set table 410.

In the example shown in FIG. 4A, a flow ID may also include a secondoffset corresponding with a priority 406 of the packet 402. As shown inFIG. 4A, the number of flow set table entries for each priority set mayvary by aggregation group. Therefore, the priority set offsets for thepacket 402 may depend on the aggregation group ID 402, as well as thepriority 406. For example, the priority set offsets for priority set 0for aggregation groups 1-3 are all 0, because the priority set 0 entriesfor each of the aggregation groups 1-3 are listed first in theirrespective portions of the flow set table 410. However, the priority setoffset for aggregation group 1, priority set 1 would be 3, the priorityset offset for aggregation group 2, priority set 1 would be 5, and thepriority set offset for aggregation group 3, priority set 1 would be 3,as may be seen in FIG. 4A. In other embodiments, the entries of the flowset table 410 may be arranged differently, such as being grouped firstby priority set and then being group by aggregation group ID.

As shown in FIG. 4B, the flow set table entry 412 may include an index414, an assigned member entry 416 and a time stamp 418 for a last packetof a data flow corresponding with the flow set entry 412. In thisexample, the aggregation group that is used to transmit the packet 402is explicitly indicated in the packet 402, as is the priority of thepacket and may be used to determine the aggregation group offset andpriority offset. Therefore, in this example, the index 414 may be anindex to a specific flow set table entry 412 within the portion of theflow set table 410 that corresponds with the aggregation group ID 404and the priority 406 indicated in the packet 402.

The assigned member entry 416 indicates a currently assigned aggregationgroup member for a corresponding data flow and the time stamp 418 is thetime stamp of the last packet that was processed for the correspondingdata flow. The time stamp entry 418 may be subtracted from a time stampincluded in a next packet of the corresponding data flow to determinewhether the data flow is active (e.g., the difference is less than orequal to an inactivity threshold) or inactive (e.g., the difference isgreater than the inactivity threshold). As previously discussed, if itis determined that the corresponding data flow is active, the aggregatemember indicated by the member assignment 416 will be used to transmitthe packet 402. However, if it is determined that the corresponding dataflow is inactive, the corresponding data flow may receive a newaggregate member assignment, which may be the same aggregate member asthe previous aggregate member assignment, or may be a differentaggregate member, depending the respective states (e.g., datacommunication loading) of the corresponding aggregation group membersthat are available for assignment.

In this example, when an aggregate member assignment is made, the memberassignment entry 416 of the flow set entry 412 for the correspondingdata flow is updated to reflect the current aggregate member assignment.Further, each time a packet 402 of the corresponding data flow isprocessed, the time-stamp entry 418 of the corresponding flow set tableentry 412 is updated using, for example, a current system time.Alternatively, the time stamp entry 418 may be updated using a timestamp provided by a network device that is processing the correspondingdata flow.

FIGS. 5A and 5B are, respectively, tables 500 and 510 illustratingquality metric quantization in accordance with an example embodiment.FIG. 5C is a diagram illustrating a quality mapping table based on thequality metrics of FIGS. 5A and 5B in accordance with an exampleembodiment. The approach illustrated in FIGS. 5A-5C may be used toassign a respective combined quality metric to each member of anaggregation group. These respective quality metrics may then be groupedinto quality bands or loading bands. In an example network device,aggregate member assignments for data flows being processed by a networkdevice may be made from a subset of aggregate members that have thequality metrics in the first non-empty quality band, or the highestquality band that includes at least one aggregate member. The qualitymetrics described with respect to FIGS. 5A-5C are given by way ofexample and other quality metrics may be used when assigning aggregatemembers for transmitting data flows using the techniques describedherein.

FIG. 5A is a table 500 in accordance with an example embodiment that maybe used to quantize port loading into a first quality metric. In thetable 500, the first column includes port loading threshold values,which may be compared with the amount data that is assigned to eachaggregate member of an aggregation group over a given period of time.The threshold values in table 500 are given by way of example. In otherembodiments, different threshold values may be used. In otherembodiments, a different number of threshold values (fewer or morethreshold values) may be used.

In the example table 500, eight port loading threshold values are used,which correspond with a three bit quantization value (binary values000-111). As shown in table 500 in FIG. 5A, port loading of less than100 kilobytes (kB), in this example, would be quantized with a portloading quality metric of 000. As the threshold values increase in table500, the corresponding quantization values for the port loading qualitymetric also increase to represent the higher loading (and lower qualitymetric). For instance, port (aggregate member) loading between 400 kBand 500 kB would be quantized with a port loading quality metric of 100.For sake of brevity, each threshold value in table 500 (and table 510)is not specifically described here.

In an example network device, port loading may be periodically measuredand compared with the threshold values in the table 500, such as usingan imbalance detection module 214 as illustrated in FIG. 2. The measuredport loading values for each member that are compared with the thresholdvalues in table 500 may indicate instantaneous loading or,alternatively, may indicate average loading over time. For instance,loading may be computed using a weighted average calculation. In otherembodiments, the table 500 could be used to quantify respective loadingquality metrics for priority sets instead of aggregate members (ports).

FIG. 5B is a table 510 in accordance with an example embodiment that maybe used to quantify data queue sizes into a second set of respectivequality metrics (e.g., for members of an aggregation group or,alternatively, for a plurality of priority sets). In the table 510, thefirst column includes queue size threshold values (e.g., in number ofcells or bytes), which may be compared with instantaneous queue sizes oraverage queue sizes (e.g., weighted average queue sizes) to determine acorresponding queue size quality metric. As with the table 500, thethreshold values in table 510 are given by way of example. In otherembodiments, different threshold values may be used. In otherembodiments, a different number of threshold values (fewer or morethreshold values) may be used.

In the example table 510, eight queue size threshold values are used,which each correspond with a respective three bit quantization value(binary values 000-111). As shown in table 510 in FIG. 5B, a queue sizeof less than 100 (e.g., 100 cells or packets), in this example, would bequantized with a queue size quality metric of 000. As with the table500, as the threshold values increase in table 510, the correspondingquantization values for the queue size quality metric also increase torepresent the larger queue sizes (and lower quality metric). Forinstance, queue size measurements between 500 and 1000 would bequantized with a queue size quality metric of 011.

In an example network device, queue sizes may be periodically measuredand compared with the threshold values in the table 510. For instance,queue size information could be provided to the imbalance detectionmodule 214 of the network device 200 by the aggregate member queues. Themeasured queue size values for each aggregate member that are comparedwith the threshold values in table 510 may indicate instantaneous queuesizes or, alternatively, may indicate average queue sizes over time, aswas noted above. In other embodiments, as also indicated above, thetable 510 could be used to quantify queue size quality metrics for a setof priority queues (priority set queues) associated with the aggregatemember.

FIG. 5C is a diagram that illustrates an approach for combining(mapping) the quality metrics described above with respect to tables 500and 510 into a single quality metric with a value between 0 and 7. Inexample embodiments, different mapping relationships for combining thosequality metrics may be used. The particular mapping arrangement maydepend on the embodiment and may, for example, be selected by a user.Alternatively, such quality metric mapping may be dynamic and adjustedover time in order to improve the efficiency of aggregate memberassignments in a network device. Such dynamic adjustment may beperformed by the user, software, or some other control mechanism. Theparticular approach shown in FIG. 5C is given by way of example. It willbe appreciated that a number of other possible mapping relationships arepossible.

As shown in FIG. 5C, when the values of the port loading quality metricof FIG. 5A and the values of the queue size quality metric of FIG. 5Bare combined and arranged in the sequence as shown in FIG. 5C, thosecombined quality metrics may be mapped to a single quality metric, asshown in FIG. 5C, designated member quality. In this example, a combinedmember quality metric of 7 represents the highest quality members, whilea combined quality metric of 0 represents the lowest quality members.For instance, when both the queue size metric and the port loadingmetric are 000, the combined quality metric maps to 7. Also, when boththe queue size quality metric and the port loading quality metric are111, the combined quality metric maps to 0. The particular mappingarrangement used to generate such a combined quality metric may bereferenced as a quality mapping function and may be accomplished in anumber of fashions, such as using a lookup table, using one or moremathematical transformations, such as comparison with a set ofthresholds, or other appropriate techniques. Depending on the particularembodiment, a quality mapping function for each aggregation group of anetwork device may be arbitrarily selected from a plurality of qualitymapping functions, such as the quality mapping function illustrated inFIG. 5C.

After respective combined quality metrics are determined for a set ofaggregate members, those combined quality metrics may be grouped intoloading/quality bands corresponding with the combined quality metrics.Such quality bands may then be used to assign aggregate members to dataflows, such as using the approaches described herein.

FIG. 6 is a graph 600 that illustrates an example of quality bandgroupings for an aggregation group having four members (Port 1-Port 4).In other embodiments, the graph 600 may represent quality band groupingsfor a single priority set. The groupings shown in the graph 600correspond with a quality metric having values in a range of 0-7, suchas the combined quality metric described above with respect to FIG. 5C.In this example, as shown in the graph 600, the quality metric 602 forPort 1 is in Quality Band 5, which may correspond with a combinedquality of metric of 5. In the table 600, the quality metrics 604 and606 for Port 2 and Port 3, respectively, are in Quality Band 4, whichmay correspond with a combined quality metric of 4. Also in the table600, the quality metric 608 is in Quality Band 0, which may correspondwith a combined quality metric of 0.

In this example, when assigning an aggregate member to a data flow(e.g., a new data flow or inactive data flow, as were described above)using the techniques described herein, a network device may examine thegroupings in the graph 600 to select the highest non-empty quality band(the highest quality band having at least one aggregate member in it),which in this case is Quality Band 5. The network device may them assignan aggregate member to the data flow from the selected quality band. Inthis case, because a single aggregate member (Port 1) is present inQuality Band 5, the selection process is straightforward and Port 1would be selected as the assigned member for the data flow for which anaggregate member assignment is being made. However, in situations wheremore than one aggregate member is present in the highest non-emptyquality band, other approaches for aggregate member selection may beused.

As some examples, an aggregate member assignment may be made by randomlyselecting an aggregate member from the aggregate members present in thehighest non-empty quality band. In other embodiments, a round-robinselection method may be used for aggregate member assignment selection.In still other embodiments, other deterministic and non-deterministicapproaches for aggregate member selection may be used. For instance, inone approach, in one approach member assignment could be weighted basedon quality metrics, such as those described herein.

FIG. 7 is a diagram illustrating an approach for random selection of anaggregate member using a binary search in accordance with an exampleembodiment. In the example shown in FIG. 7, a member assignmentselection is made from a set of eight aggregate members, where 5 ofthose members are present in a selected quality band (which may or maynot be a highest, non-empty quality band). In the example shown in FIG.7, the aggregate members present in the selected quality band are shownusing an 8-bit selected member vector. The members present in thequality band are represented by a ‘1’ in the vector, while the membersnot present are represented by a ‘0’ in the vector.

In this example, the value of the selected member vector is assumed, forpurposes of illustration, to have a binary value of ‘10110110.’ Thisvalue may directly represent the aggregate members present in theselected quality band or could be produced by shifting an initial vectorin order to increase randomness of aggregate member selection. Othertechniques for increasing selection randomness may also be employed.

Because the loading band vector in this example is 8-bits in length, atotal of three binary search stages (iterations) are used to select aneligible member. In this example, a random three bit selection value mayalso be generated, where the random selection value is used to breakties that may occur at each binary search stage. A tie occurs wheneligible members exist to both the left and right of the binary searchpoint. In this example, the selection value is given as binary ‘011.’

In each stage, the selected member vector is split into two halvestermed an upper vector and a lower vector. In the first stage shown inFIG. 7, both the upper vector (1011) and lower vector (0110) have aneligible member. In this situation, the most significant bit (MSB) ofthe random selection value may be used to determine which vector isselected. Since the MSB of the random selection value is ‘0’, in thisexample, the lower vector (to the right of the split) is selected andthe MSB of the Index/Member ID is set to ‘0’. In other embodiments, a‘0’ in the corresponding selection value bit location may indicate thatthe upper vector (to the left of the split) is to be selected. However,in this example, ‘0’ indicates that the lower vector should be selectedand ‘1’ indicates that the upper vector should be selected.

In the second stage of the binary search illustrated in FIG. 7, thevector selected in the previous stage (0110) is again split into a lowervector (10) and an upper vector (01). Again, both vectors containeligible members, thus the middle bit (bit 1) of the random selectionvalue is used to break the tie. Because bit 1 of the random selectionvalue is ‘1’, in this example, the upper vector (01) is selected and bit1 of the Index/Member ID is set to ‘1’.

In the third stage of the binary search illustrated in FIG. 7, thevector selected in prior round (01) is split again into an upper vector(0) and a lower vector (1). Because only the lower vector contains aneligible member, it is selected (the least significant bit (LSB), bit 0,of the random selection value is not used) and the LSB of the member IDis set to ‘0’ result in an index of ‘010’.

In embodiments where an initial vector is shifted to increase randomnessof eligible member selection, in order to identify the correct member inthe loading band occupancy set before the shift operation, a valuecorresponding with the shift must be added must be added to the memberID determined by the binary search to correct for the shift. Forexample, if a right shift of three was used in the example shown in FIG.7 (i.e., to shift a vector of B“10110101” to the vector B“10110110”noted above), then a value of 3 must be added to the determined memberID to correct for the shift. Therefore, in this example, after adding 3to the member ID determined by the binary search, the corrected randomlyselected member ID is 5. It will be appreciated that a number of othermember selection techniques may be used to select an eligible aggregatemember from a selected quality band having a plurality of eligiblemembers present and that the foregoing described techniques are given byway of example.

FIGS. 8-12, 14 and 15 are flowcharts that illustrate methods that may beused to implement the load balancing techniques described herein. Itwill be appreciated that these methods may be implemented using thetechniques described with respect to the other FIGs. of the application,or may be combined with each other in appropriate combinations. Further,for each of the methods described herein, the method operations may beperformed in any appropriate order. For instance, in some situations,certain operations may be performed concurrently, or may be performed ina different order than illustrated and described herein.

FIG. 8 is a flowchart illustrating a method 800 for dynamic loadbalancing in accordance with an example embodiment. The method 800 maybe implemented, for example, in the network device 200 illustrated inFIG. 2. As illustrated in FIG. 8, the method 800, at block 805 includesreceiving, e.g., at a network device, a data packet to be sent via anaggregation group, where the aggregation group comprises a plurality ofaggregate members. At block 810, the method 800 includes determining,based on the packet, a flow ID, such as using the approaches describedabove.

The method 800 may further include (though not explicitly shown in FIG.8) indexing a flow set table (such as the flow set table 412) using theflow ID, for instance, in the manner described above. At block 815, themethod 800 includes determining a state of the flow, such as by using atimestamp included in the flow set table and a timestamp associated withthe current packet, or by determining that a corresponding entry in theflow set table is not valid, such as using the techniques describedabove. At block 820, the method 800 may include determining, based onthe flow identifier and the state of the flow, an assigned member of theplurality of aggregate members for the flow, such as using qualitymetrics and associated quality bands, as was previously described. Atblock 825, the method 800 includes communicating the packet via theassigned member.

FIG. 9 is a flowchart illustrating a method for indexing a flow settable in accordance with an example embodiment. The method 900 includes,at block 905, determining a first offset based on an identifier for theaggregation group included in the packet. The method 900 furtherincludes, at block 910, generating a hash value from the packet, such asby using a fixed set of packet fields and a hash function (e.g., a CRChash function or an XOR hash function). At block 915, the method 900includes determining a priority of the packet and, at block 920,determining a second offset based on the priority. In the method 900, atblock 925, the first offset may be used as a pointer for indexing aportion of the flow set table corresponding with the aggregation group.At block 930, the method 900 may include using the second offset as apointer for indexing a sub-portion of the flow set table correspondingwith the aggregation group, where the sub-portion corresponds with thedetermined priority and the aggregation group indicated in the packet.The method 900 further includes, at block 935, using the hash value toindex a flow set table entry in the sub-portion of the tablecorresponding with the aggregation group and the determined priority.

FIG. 10 is a flowchart illustrating a method 1000 for determining aquality measure for a given aggregate member in accordance with anexample embodiment. The method 1000 may be used to determine a qualitymeasure using, for example, the techniques described above with respectto FIGS. 5A-5C, though other approaches exist. The method 1000, at block1005 includes periodically measuring average data traffic loading forthe given aggregate member, such as using the imbalance detection module214 illustrated in FIG. 2. The method 1000 further includes, at block1010, quantizing the average data traffic loading to a binary loadingmetric. The quantization at block 1010 may be accomplished using theaverage traffic loading and the table 500 illustrated in FIG. 5A. Themethod 1000 may further include periodically measuring an average queueddata amount for the given aggregate member, such as using the aggregatemember queues 212 and/or and the imbalance detection module 214, asdiscussed above.

The method 1000 may further include, at block 1020, quantizing theaverage queued data to a binary queue size metric. The quantization atblock 1020 may be accomplished using the average queued data amountmeasurement and the table 510 illustrated in FIG. 5B. At block 1025, themethod 1000 includes combining the binary loading metric and the binaryqueue size metric to produce a combined quality metric, such as usingthe techniques illustrated in FIG. 5C. For instance, such a combinedquality metric may be a concatenation of the quality metrics determinedat blocks 1010 and 1020. At block 1030, the method 1000 further includesmapping the combined quality metric to a respective quality measureusing a quality mapping function, as was also discussed with respect toFIG. 5C.

FIG. 11 is a flowchart illustrating a method 1100 for generatingquality/loading bands for an aggregation group and selecting anaggregate member using the quality/loading bands in accordance with anexample embodiment. The method 1100 includes, at block 1105, groupingrespective quality measures for each member of an aggregation group(e.g., that are determined at block 1030 for members of the aggregationgroup) into loading/quality bands. Each loading/quality band may have arespective upper threshold and a respective lower threshold, wherein theloading/quality bands are contiguous and non-overlapping. For instance,the groupings may be implemented using the techniques illustrated inFIG. 6, where the quality bands correspond with the combined respectivequality metrics determined at block 1030 of the method 1000 and thethresholds correspond with the values of the respective quality metrics.

The method 1100, in this example, may include, at block 1110, generatinga random number, and at block 1115, performing a binary search of abit-map representing the members of the aggregation group included inthe quality/loading band with the largest upper and lower thresholdsthat includes one or more members of the aggregation group (e.g.,eligible members). In such an approach, the random number may be used toselect a direction for the binary search, such as in the mannerdescribed above with respect to the random selection value illustratedin FIG. 7. In certain embodiments, in order to be an eligible member ofa quality/loading band, an aggregate member must be available for datatransmission. For instance, links that are down would not be included inquality/loading bands as eligible members.

FIG. 12 is a flowchart illustrating a method 1200 for dynamic loadbalancing for virtual links of an aggregation group in accordance withan example embodiment, where transmission credits are used as a measureof link quality. In such embodiments, a number available transmissioncredits may be used as a measure of aggregate member quality instead ofthe quality metrics described above with respect to FIGS. 5A-5C. Usingsuch an approach, members of an aggregation may be grouped in qualitybands based on their respective number of available transmissioncredits.

The method 1200 includes, at block 1205 assigning, e.g., at a networkdevice, an aggregation group transmission credit quota for anaggregation group, where the aggregation group comprises a plurality ofvirtual links. At block 1210, the method 1200 includes assigningrespective amounts of transmission credits of the aggregation grouptransmission credit quota to each virtual link of the aggregation group.The method also includes, at block 1215 receiving a data packet that isto be communicated using the aggregation group. At block 1220, themethod 1200 includes determining a hash value based on the packet, suchas by using the techniques described herein.

The method 1200 further includes, at block 1225, determining an assignedvirtual link of the plurality of virtual links based on the hash value(e.g., using a flow set table and the techniques described above). Themethod 1200 also includes, at block 1230, reducing the aggregation grouptransmission credit quota by an amount proportional to the size of thepacket and, at block 1235, reducing the respective amount oftransmission credits assigned to the assigned virtual link by the amountproportional to the size of the packet. At block 1240, the method 1240includes communicating the packet to another network device using theassigned virtual link.

In the method 1200, the plurality of virtual links may comprise aplurality of equal-cost-multiple-path (ECMP) links. Alternatively, theplurality of virtual links may comprise a plurality of weighted-costpaths. Further in the method 1200, the plurality of virtual links maycorrespond with a single physical link. Alternatively in the method1200, the plurality of virtual links may be implemented using a linkaggregation group (LAG) or a HiGig trunk, as two examples.

Further in the method 1200, reducing the number of availabletransmission credits for the aggregation group may comprise reducing thenumber of available transmission credits for the aggregation group by anamount proportional to the size of the packet (e.g., an observed size,as received, or the size of the packet after it has been encapsulated ordecapsulated, depending on the specific implementation). Likewise,reducing the number of available transmission credits for the assignedvirtual link comprises reducing the number of available transmissioncredits for the assigned virtual link by the amount proportional to thesize of the packet (e.g., observed (received) size, or afterencapsulation/decapsulation).

In another example embodiment, reducing the number of availabletransmission credits for the aggregation group at block 1230 maycomprise reducing the number of available transmission credits for theaggregation group by one transmission credit, where each packettransmitted results in one transmission credit being deducted,regardless of the size of the packet. Further, in like fashion, in thisexample embodiment, reducing the number of available transmissioncredits for the assigned virtual link may comprise reducing the numberof available transmission credits for the assigned virtual link by onetransmission credit.

In the method 1200, reducing the number of available transmissioncredits for the aggregation group at block 1230 may comprise reducingthe number of available transmission credits for the aggregation groupprior to communicating the packet to the other network device. Likewise,reducing the number of available transmission credits for the assignedvirtual link at block 1235 may comprise reducing the number of availabletransmission credits for the assigned virtual link prior tocommunicating the packet to the other network device.

In an example embodiment, a sum of the respective amounts oftransmission credits that are assigned to the plurality of virtual linksat block 1210 equals the assigned aggregation group transmission creditquota of block 1205. Using such an approach, the respective amounts oftransmission credits assigned to each of the plurality of virtual linksmay be equal or may be weighted.

As indicated above, the amount of available transmission credits may beused a measure of member quality. The aggregate members for a giveaggregation group may be grouped into quality bands based on theirrespective number of available transmission credits. Using such anapproach, determining an assigned virtual link of the plurality ofvirtual links would be based on the respective amounts of availabletransmission credits for each of the plurality of virtual links (e.g.,as represented by corresponding quality bands for the aggregate membersbased on their respective available transmission credits).

FIG. 13 is a diagram illustrating an embodiment of quality/loading bandsin accordance with an example embodiment. In the diagram of FIG. 13, thequality/loading bands are based on a number of available transmissioncredits. As shown in FIG. 13, quality bands are displayed for four linksLink 1-Link 4, which may be virtual links, for example. Alternatively,Link 1-Link 4 may comprise physical links.

The graph shown in FIG. 13 includes four contiguous, non-overlappingquality/loading bands A-D. The Loading Bands A-C (Upper Band) includeaggregate members that have a surplus (available) transmission credits.The Loading Band D (Lower Band) includes aggregate members that have adeficit of transmission credits (negative transmission credit balances).As shown in FIG. 13, each aggregate member may only accumulate a surplusof transmission credits that is less than or equal to a transmissioncredit ceiling and, also, may not accumulate a deficit transmissionbalance that is less than a transmission credit floor.

In the graph shown in FIG. 13, the Loading Band A would includeaggregate members having the highest member quality, while the LoadingBand D would include aggregate members having the lowest member quality.The Link 1 in FIG. 13 has a surplus transmission credit balance 1302 andis a member of Loading Band C. The Link 2 has a deficit transmissioncredit balance 1304 and is a member of Loading Band D. The Links 3 and 4in FIG. 3 have respective surplus transmission credit balances 1306 and1308 and are both members of Loading Band B. In the graph shown in FIG.13, Loading Band A does not have any eligible members. Thus, in anexample embodiment, if an aggregate member assignment is made using thequality/loading bands shown in FIG. 13, it would be made from LoadingBand B with a member selection vector ‘0011’. An assigned link may bedetermined from the member selection vector using number of techniques,such as the techniques discussed above.

FIG. 14 is a flowchart illustrating a method 1400 for generatingquality/loading bands for an aggregation group and selecting anaggregate member using the quality/loading bands in accordance with anexample embodiment. The method 1400, at block 1405 includes assigning(e.g., at a network device) an aggregation group transmission creditquota to an aggregation group, the aggregation group including aplurality of members. The method 1400 further includes, at block 1410,assigning respective amounts of transmission credits of the aggregationgroup transmission credit quota to each member of the aggregation group.At block 1415, the method 1400 includes reducing the aggregation grouptransmission quota and the respective amounts of transmission credits oftransmitting members of the aggregation group in correspondence witheach packet transmitted by the aggregation group, such as using thetechniques described herein.

At block 1420, the method 1400 includes grouping the plurality ofmembers into loading/quality bands based on their respective amount ofavailable transmission credits. In the method 1400, each loading/qualityband may have a respective upper threshold and a respective lowerthreshold, and the loading/quality bands may be contiguous andnon-overlapping. Further, the method 1400 includes, at block 1425,selecting members of the aggregation group for transmitting packets fromthe quality/loading band with the largest upper and lower thresholdsthat includes one or more members of the aggregation group, such asusing the approach described above with respect to FIG. 13.

In the method 1400, reducing the aggregation group transmission quotaand the respective amount of transmission credits of the transmittingmember at block 1415 may comprise reducing the aggregation grouptransmission quota and the respective amount of transmission credits ofthe transmitting member by respective amounts of transmission creditscorresponding with each transmitted packet. Alternatively, reducing theaggregation group transmission quota and the respective amount oftransmission credits of the transmitting member at block 1415 of themethod 1400 may comprise reducing the aggregation group transmissionquota and the respective amount of transmission credits of thetransmitting member by one transmission credit for each transmittedpacket.

In the method 1400, reducing the aggregation group transmission quotaand the respective amounts of transmission credits of transmittingmembers of the aggregation group at block 1415 may comprise reducing theaggregation group transmission quota and the respective amounts oftransmission credits of transmitting members of the aggregation groupupon transmission of each packet transmitted by the aggregation group.Alternatively, in the method 1400, reducing the aggregation grouptransmission quota and the respective amounts of transmission credits oftransmitting members of the aggregation group at block 1415 may comprisereducing the aggregation group transmission quota and the respectiveamounts of transmission credits of transmitting members of theaggregation group prior to transmission of each packet transmitted bythe aggregation group.

In the method 1400, the plurality of members may comprise a plurality ofphysical links. Alternatively, the plurality of members comprises aplurality of virtual links.

FIG. 15 is a flowchart illustrating a method 1500 for generatingquality/loading bands for an aggregation group and selecting anaggregate member using the quality/loading bands in accordance with anexample embodiment. The method 1500 includes, at block 1505, determining(e.g., by a network device) respective quality metrics for each of aplurality of members of an aggregation group of the network device. Inthe method 1500, the respective quality metrics may represent respectivedata traffic loading for each member of the aggregation group, such asdescribed above with respect to FIGS. 5A-5C, for example.

The method 1500 may further include, at block 1510, grouping theplurality of aggregation members into a plurality of loading/qualitybands based on their respective quality metrics, such as was illustratedin FIG. 6, for example. At block 1515, the method 1500 includesselecting members of the aggregation group for transmitting packets froma loading/quality band corresponding with members of the aggregationgroup having lower data traffic loading relative to the other members ofthe aggregation group, e.g., by selecting an aggregate member from ahighest non-empty loading/quality band.

In the method 1500, the respective quality metrics may be based on atleast one of port loading corresponding with each aggregate member andan amount of queued data corresponding with each aggregate member. Forinstance, each respective quality metric may be determined by combininga first quality metric corresponding with respective port loading and asecond quality metric corresponding with the respective amount of queueddata, such as was described above with respect to FIGS. 5A-5C. In otherapproaches, a number of available transmission credits may be used as ametric for determining link quality.

In the method 1500, selecting members of the aggregation group at block1515 may comprise randomly selecting members of the aggregation group.Alternatively, selecting members of the aggregation group at block 1515may comprise deterministically selecting members of the aggregationgroup, such as using round-robin selection, for example.

Implementations of the various techniques described herein may beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or in combinations of them. Implementations mayimplemented as a computer program product, i.e., a computer programtangibly embodied in an information carrier, e.g., in a machine-readablestorage device, for execution by, or to control the operation of, dataprocessing apparatus, e.g., a programmable processor, a computer, ormultiple computers. A computer program, such as the computer program(s)described above, can be written in any form of programming language,including compiled or interpreted languages, and can be deployed in anyform, including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program can be deployed to be executed on one computer or onmultiple computers at one site or distributed across multiple sites andinterconnected by a communication network.

Method steps may be performed by one or more programmable processorsexecuting a computer program to perform functions by operating on inputdata and generating output. Method steps also may be performed by, andan apparatus may be implemented as, special purpose logic circuitry,e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Elements of a computer may include atleast one processor for executing instructions and one or more memorydevices for storing instructions and data. Generally, a computer alsomay include, or be operatively coupled to receive data from or transferdata to, or both, one or more mass storage devices for storing data,e.g., magnetic, magneto-optical disks, or optical disks. Informationcarriers suitable for embodying computer program instructions and datainclude all forms of non-volatile memory, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in special purposelogic circuitry.

To provide for interaction with a user, implementations may beimplemented on a computer having a display device, e.g., a cathode raytube (CRT) or liquid crystal display (LCD) monitor, for displayinginformation to the user and a keyboard and a pointing device, e.g., amouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input.

Implementations may be implemented in a computing system that includes aback-end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront-end component, e.g., a client computer having a graphical userinterface or a Web browser through which a user can interact with animplementation, or any combination of such back-end, middleware, orfront-end components. Components may be interconnected by any form ormedium of digital data communication, e.g., a communication network.Examples of communication networks include a local area network (LAN)and a wide area network (WAN), e.g., the Internet.

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the true spiritof the embodiments of the invention.

1. A method comprising: assigning, at a network device, an aggregationgroup transmission credit quota to an aggregation group, the aggregationgroup including a plurality of members; assigning respective amounts oftransmission credits of the aggregation group transmission credit quotato each member of the aggregation group; reducing the aggregation grouptransmission quota and the respective amounts of transmission credits oftransmitting members of the aggregation group in correspondence witheach packet transmitted by the aggregation group; grouping the pluralityof members into quality/loading bands based on their respective amountof available transmission credits, each quality/loading band having arespective upper threshold and a respective lower threshold, wherein thequality/loading bands are contiguous and non-overlapping; and selectingmembers of the aggregation group for transmitting packets from thequality/loading band with a largest upper threshold and a largest lowerthreshold that includes one or more members of the aggregation group. 2.The method of claim 1, wherein reducing the aggregation grouptransmission quota and the respective amount of transmission credits ofthe transmitting member comprises reducing the aggregation grouptransmission quota and the respective amount of transmission credits ofthe transmitting member by respective amounts corresponding with eachtransmitted packet.
 3. The method of claim 1, wherein reducing theaggregation group transmission quota and the respective amount oftransmission credits of the transmitting member comprises reducing theaggregation group transmission quota and the respective amount oftransmission credits of the transmitting member by one for eachtransmitted packet.
 4. The method of claim 1, wherein selecting membersof the aggregation group comprises: generating a random number; andperforming a binary search of a bit-map representing the members of theaggregation group included in the quality/loading band with the largestupper threshold and the largest lower threshold that includes one ormore members of the aggregation group, wherein the random number is usedto select a direction for the binary search.
 5. The method of claim 1,wherein the plurality of members comprises a plurality of physicallinks.
 6. The method of claim 5, wherein reducing the aggregation grouptransmission quota and the respective amounts of transmission credits oftransmitting members of the aggregation group comprises reducing theaggregation group transmission quota and the respective amounts oftransmission credits of transmitting members of the aggregation groupupon transmission of each packet transmitted by the aggregation group.7. The method of claim 1, wherein the plurality of members comprises aplurality of virtual links.
 8. The method of claim 7, wherein reducingthe aggregation group transmission quota and the respective amounts oftransmission credits of transmitting members of the aggregation groupcomprises reducing the aggregation group transmission quota and therespective amounts of transmission credits of transmitting members ofthe aggregation group prior to transmission of each packet transmittedby the aggregation group.
 9. The method of claim 1, wherein thequality/loading bands comprise a plurality of positive balancequality/loading bands and a negative balance quality/loading band.
 10. Anetwork device comprising: at least one processor; a non-transitorymachine-readable medium having instructions stored thereon, wherein theinstructions, when executed by the at least one processor, cause thenetwork device to implement: an imbalance detection module configuredto: assign, an aggregation group transmission credit quota to anaggregation group of the network device, the aggregation group includinga plurality of members; assign respective amounts of transmissioncredits of the aggregation group transmission credit quota to eachmember of the aggregation group; and reduce the aggregation grouptransmission quota and the respective amounts of transmission credits oftransmitting members of the aggregation group in correspondence witheach packet transmitted by the aggregation group; and an aggregatemember selection module coupled with the imbalance detecting module, theaggregate member selection module being configured to: group theplurality of members into quality/loading bands based on theirrespective amount of available transmission credits, eachquality/loading band having a respective upper threshold and arespective lower threshold, wherein the quality/loading bands arecontiguous and non-overlapping; and select members of the aggregationgroup for transmitting packets from the quality/loading band with alargest upper threshold and a largest lower threshold that includes oneor more members of the aggregation group.
 11. The network device ofclaim 10, wherein the plurality of members comprises a plurality ofphysical links.
 12. The network device of claim 10, wherein theplurality of members comprises a plurality of virtual links.
 13. Thenetwork device of claim 10, wherein the aggregate member selectionmodule is configured to select members of the aggregation group by:generating a random number; and performing a binary search of a bit-maprepresenting the members of the aggregation group included in thequality/loading band with the largest upper threshold and the largestlower threshold that includes one or more members of the aggregationgroup, wherein the random number is used to select a direction for thebinary search.
 14. The network device of claim 10, wherein reducing theaggregation group transmission quota and the respective amount oftransmission credits of the transmitting member comprises reducing theaggregation group transmission quota and the respective amount oftransmission credits of the transmitting member by one for eachtransmitted packet.
 15. The network device of claim 11, wherein reducingthe aggregation group transmission quota and the respective amounts oftransmission credits of transmitting members of the aggregation groupcomprises reducing the aggregation group transmission quota and therespective amounts of transmission credits of transmitting members ofthe aggregation group upon transmission of each packet transmitted bythe aggregation group.
 16. The network device of claim 12, whereinreducing the aggregation group transmission quota and the respectiveamounts of transmission credits of transmitting members of theaggregation group comprises reducing the aggregation group transmissionquota and the respective amounts of transmission credits of transmittingmembers of the aggregation group prior to transmission of each packettransmitted by the aggregation group.
 17. The network device of claim10, wherein the quality/loading bands comprise a plurality of positivebalance quality/loading bands and a negative balance quality/loadingband.